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Abstract 

This  paper  describes  a  research  program  that  is  developing  more  robust  algorithms  for  The 
Improved  Performance  Research  Integration  Tool  (IMPRINT)  that  better  reflect  the  complexity 
of  the  effects  of  training  on  human  performance.  The  Air  Force  Research  Laboratory’s  War¬ 
fighter  Readiness  Research  Division  is  sponsoring  a  series  of  empirical  studies  to  assess  the  ef¬ 
fects  of  various  training  strategies  on  skill  acquisition  and  retention  in  the  performance  of  com¬ 
plex  military  tasks.  New  algorithms  will  be  developed  and  populated  based  on  the  results  of 
these  studies.  Ongoing  research  efforts  are  described.  The  relevance  of  this  research  for  system 
designers  is  discussed. 

The  Improved  Performance  Research  Integration  Tool 

The  Improved  Performance  Research  Integration  Tool  (IMPRINT)  is  a  modeling  and  si¬ 
mulation  software  package  that  is  used  by  system  designers  to  evaluate  human  system  integra¬ 
tion  (HSI)  questions  throughout  the  life-cycle  of  a  system.  This  tool  can  be  used  both  for  the  hu¬ 
man  engineering  of  a  system  and  for  the  estimation  of  certain  manpower  costs  across  the  sys¬ 
tem's  life  cycle.  Using  task  networks.  IMPRINT  is  used  to  estimate  human-system  limitations 
and  performance  concerns,  manpower  needs,  and  task  allocation  effects  on  workload  and  per¬ 
formance.  IMPRINT  also  allows  system  designers  to  estimate  the  performance  impacts  of  tem¬ 
perature  variations,  vibration,  fatigue,  and  stressors.  Use  of  this  tool  during  development  affords 
the  identification  of  human  factors  engineering  issues,  fostering  human-centered  system  design 
and  enhanced  human  and  system  performance. 

IMPRINT  currently  addresses  two  requirements:  (1)  estimating  how  human  performance 
impacts  system/mission  success,  and  (2)  estimating  how  maintenance  personnel's  individual  and 
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team  performance  over  sustained  periods  can  affect  mission  effectiveness  across  units  and  at  the 
force  level.  Most  of  the  human  performance  variables  embedded  in  IMPRINT  have  been  devel¬ 
oped  and  validated  using  empirical  research  findings.  The  current  IMPRINT  training  algorithms, 
however,  are  based  on  limited  experimental  data.  The  only  training-related  performance  effect 
available  to  a  modeler  is  the  time  period  since  the  last  training  event  (e.g.,  last  week,  one  month 
ago,  six  months  ago).  Task  performance  times  increase  and  task  accuracy  decreases  with  less 
frequent  training 

The  primary  goal  of  the  IMPRINT  Training  Algorithm  Enhancement  (ITAE)  Program  is 
to  include  consideration  of  training  and  its  impacts,  especially  since  Reserve  and  Guard  units  are 
playing  an  increasingly  important  role  in  integrated  combat  teams.  Three  aspects  of  training  need 
to  be  considered: 

1 .  Initial  training  and  its  impact  on  increasing  performance  accuracy  and  reducing  perfor¬ 
mance  time, 

2.  Knowledge  and  skill  retention  over  periods  of  non-use,  and 

3.  Rate  of  recovery  wrhcn  knowledge  and  skill  is  rehearsed,  exercised,  or  re-used. 

To  achieve  this  goal,  The  Air  Force  Research  Laboratory's  Warfighter  Readiness  Research 
Division  is  sponsoring  a  series  of  empirical  studies  to  assess  the  effects  of  various  training 
strategies  on  skill  acquisition  and  retention  in  the  performance  of  complex  tasks.  In  a  recently 
completed  study,  new  algorithms  were  developed  to  reflect  the  effects  of  overtraining  on  per¬ 
formance  (McDermott,  et.  al.,  2007).  Other  studies,  which  will  be  discussed  in  greater  detail  in 
this  paper,  are  currently  in  progress  to  investigate  additional  training  effects.  Specifically,  one 
study  is  assessing  the  performance  effects  of  varying  levels  of  instructor  interaction  for  a  set  of 
aviation  and  sensor  manipulation  tasks.  In  another  effort,  researchers  are  studying  automotive 
maintenance  tasks  and  the  performance  effects  of  using  hands-on  practice  during  training.  Fi¬ 
nally,  the  efficacy  of  team  training  interventions  is  also  under  investigation. 

Instructional  Strategies 

Among  the  many  factors  impacting  overall  human  performance  on  complex  systems  is  train¬ 
ing  effectiveness,  which  can  vary  as  a  function  of  the  strategy  employed  in  training  operators, 
maintainers,  and  teams.  The  term  “instructional  strategy”  can  encompass  a  number  of  attributes 
associated  with  a  given  training  approach.  The  development  of  effective  instruction  is  driven  by 
one  of  many  underlying  instructional  theories.  For  instance,  proponents  of  the  constructivist  the¬ 
ory  of  instruction  believe  that  students  are  highly  motivated  to  learn  and,  when  provided  the  ap¬ 
propriate  resources,  students  will  explore  and  learn  the  relevant  skills.  The  primary  role  of  the 
instructor  is  to  provide  a  rich  learning  environment  and  to  serve  as  a  facilitator  for  the  students. 
Depending  upon  the  theoretical  orientation  of  the  instructional  designer,  a  variety  of  instructional 
methods  and  media  may  be  used  to  facilitate  learning.  Keeping  with  our  example  of  constructiv¬ 
ist  theory,  a  proponent  would  likely  provide  learners  w'ith  an  environment  that  includes  access  to 
written  texts,  team  learning  opportunities,  and  demonstrations  or  simulations  to  facilitate  the  de¬ 
velopment  of  the  skills  to  be  acquired.  Finally,  the  location,  frequency,  and  duration  of  training 
would  be  critical  considerations  for  the  efficacy  of  the  instruction.  Training  based  on  constructiv¬ 
ist  theory  would  likely  take  place  in  an  environment  in  which  learners  can  interact  with  peers  and 
have  hands-on  experience  with  actual  equipment  or  simulations  of  a  system  of  interest,  rather 
than  in  a  classroom  setting.  With  so  many  parameters,  the  number  of  potential  instructional 
strategies  employed  to  train  an  operator  is  nearly  limitless.  A  variety  of  these  instructional  strate- 


gies  are  under  investigation  in  the  studies  being  conducted  to  develop  IMPRINT  algorithms  for 
the  ITAE  program. 


IMPRINT  Training  Algorithm  Development  for  Operators 

Background 

A  tool  that  provides  insights  into  the  relative  effectiveness  of  various  training  strategies  for 
operators  of  a  given  system  could  be  highly  valuable  to  the  training  developer  seeking  to  opti¬ 
mize  human-system  performance.  The  goal  of  this  study  is  to  begin  to  quantify  the  effects  of 
various  instructional  strategies  on  operator  performance  and  to  subsequently  create  training  ef¬ 
fect  algorithms  that  can  be  implemented  within  an  IMPRINT  environment.  Such  a  capability 
would  give  the  IMPRINT  modeler  the  ability  to  predict  how  changes  in  instructional  strategies 
might  impact  the  time  and  accuracy  of  tasks  being  modeled. 

As  a  first  step  in  scoping  the  issue,  we  considered  a  dimension  of  instructional  design  that 
is  at  the  heart  of  many  cost-benefit  trade-offs.  This  dimension  is  the  level  of  instructor-student 
interaction.  In  our  schools,  low  student-teacher  ratios  are  thought  to  be  desirable  as  they  allow 
for  more  student-teacher  interaction.  However,  these  low  ratios  are  also  associated  with  higher 
costs  in  terms  of  more  classrooms  and  more  teachers.  Another  training  approach  used  with  in¬ 
creasing  frequency  is  computer-based  training  (CBT),  which  minimizes  instructional  costs  while 
essentially  eliminating  any  instructor-student  interaction.  There  is  clearly  a  financial  cost  associ¬ 
ated  with  increased  instructor-student  interaction.  Our  goal  is  to  quantify  the  benefit  (if  any)  of 
such  interaction  to  performance. 

Quantify  ing  effectiveness  of  the  training  strategies  for  given  task  types  also  poses  a  chal¬ 
lenge,  as  the  term  “effectiveness”  can  involve  a  number  of  different  parameters  with  regard  to 
training.  For  example,  one  could  consider  the  strategy  that  leads  to  the  fastest  acquisition  of  req¬ 
uisite  skills  as  being  the  most  effective.  Another  view  is  that  effectiveness  is  judged  by  the  level 
of  performance  that  can  ultimately  be  achieved  with  a  given  instructional  strategy.  Yet  another 
view  might  suggest  that  the  instructional  strategy  that  leads  to  the  highest  skill  retention  rate  over 
time  is  the  most  effective  strategy.  In  most  cases,  all  three  are  of  interest  to  a  training  system  de¬ 
veloper.  For  the  purposes  of  this  study,  we  are  interested  in  effectiveness  with  regard  to  both 
skill  acquisition  and  decay  as  measured  by  operator  task  time  and  accuracy. 

Method 

This  study  employs  a  3x2x5  mixed  design.  The  treatments  are  three  training  strategies 
(minimally,  moderately,  and  highly  instructor  interactive),  two  retention  intervals  (30  and  60 
days)  and  five  tasks.  Table  1  characterizes  the  three  training  strategies  employed. 

Sixty  (60)  college  students  are  participating  in  the  study,  with  twenty  (20)  being  random¬ 
ly  assigned  to  each  training  strategy  group.  Within  each  training  strategy  group,  half  of  the  stu¬ 
dents  were  assigned  to  a  thirty  (30)  day  retention  condition  and  half  to  a  sixty  (60)  day  retention 
condition. 

Upon  entering  the  study,  students  receive  training  on  a  series  of  five  Unmanned  Aerial 
System  (UAS)  tasks.  The  form  of  this  training  varies  across  groups,  as  shown  in  Table  1 .  The 
goal  of  the  training,  which  focuses  primarily  on  sensor  operator  duties,  is  to  enable  the  partici¬ 
pants  to  perform  these  UAS  tasks  proficiently  on  a  desktop  UAS  simulator.  After  completing  the 
academic  portion  of  the  training,  participants  perform  a  series  of  practice  trials  for  each  of  the 


five  UAS  tasks  until  they  reach  a  level  of  proficiency.  For  each  practice  trial,  task  time  and  accu¬ 
racy  data  are  recorded.  After  thirty  (30)  or  (60)  days,  depending  on  the  retention  condition  to 
which  they  are  assigned,  participants  return  and  perform  additional  simulator-based  trials  for 
each  UAS  task  and  time/accuracy  data  are  again  recorded. 


Table  1:  Three  Levels  of  Instructor-Student  Interaction  as  Implemented  in  the  Study 


Academic  Phase 

Simulator  Practice  Phase 

Minimally  Interac¬ 
tive  Condition 

CBT  with  part-task  exercises 

Instructor  available  to  answer 
only  system  operation  questions 

Moderately  Interac¬ 
tive  Condition 

Classroom  lecture  with  part-task 
exercises 

Instructor  available  to  provide 
task  advice  if  asked 

Highly  Interactive 
Condition 

Classroom  lecture  with  part-task 
exercises,  memory  probes,  and 
checks  for  understanding 

Instructor  monitors  perfor¬ 
mance  and  provides  trial-to- 
trial  task  advice  and  coaching 

The  data  resulting  from  this  design  will  allow  1TAE  researchers  to  examine  the  shape  of 
learning  curves  for  given  task  types  and  how  these  curves  might  vary  as  a  function  of  instructor- 
student  interaction.  Further,  researchers  will  be  able  to  compare  the  degree  to  which  instructor- 
student  interaction  during  training  may  impact  the  rate  at  which  skills  decay  over  a  period  of 
non-use.  Once  these  data  are  converted  to  algorithms  and  implemented  in  an  IMPRINT  envi¬ 
ronment,  we  will  have  achieved  a  first-step  in  providing  training  system  designers  a  modeling 
tool  to  aid  in  training  system  design  decisions. 

IMPRINT  Training  Algorithm  Development  for  Teams 

Background 

A  recent  review  of  the  team  performance  and  team  training  literature  (Crabtree,  Doyal,  & 
Salas,  2008)  indicated  that  considerable  research  has  been  conducted  to  identify:  the  characteris¬ 
tics  of  effective  teams,  the  necessary  teamwork  skills,  the  required  training  strategies,  and  the 
assessment  of  the  effectiveness  of  those  training  strategies.  Team  training  theorists  have  devel¬ 
oped  a  large  number  of  team  performance  and  team  training  conceptual  models  to  serve  as 
frameworks  on  which  to  build  training  concepts  and  approaches.  Recent  studies  emphasize  the 
importance  of  distinguishing  between  task-related  skills  and  teamwork  skills,  and  understanding 
team  processes  and  their  contribution  to  team  performance.  Unfortunately,  because  of  the  enor¬ 
mous  diversity  of  the  conceptual  models  and  a  lack  of  validated  guidelines,  there  is  little  stan¬ 
dardization  across  resulting  training  concepts,  especially  with  regard  to  content,  delivery,  and 
assessment  of  both  processes  and  outcomes  (van  Berio,  Lowyck,  &  Schaafstal.  2007).  Thus,  the 
results  of  team  training  interventions  can  rarely  be  compared,  and  the  task  of  determining  a 
"meanv  effect  of  a  given  strategy  is  practically  impossible.  Furthermore,  a  meta-analysis  of  more 
than  1200  team  studies  revealed  that  only  nine  provided  sufficient  data  to  enable  researchers  to 
conclude  that  three  team  training  strategies  actually  affect  teamwork  outcomes.  These  strategies 
are  team  coordination  training  (TCT),  team  cross  training  (CT),  and  team  guided  self-correction 
training  (GSC). 


The  ITAE  Program’s  investigation  into  team  training  strategy  effects  will  develop  the  capa¬ 
bility  within  IMPRINT  to  represent  the  effects  of  team  training  interventions  on  team  perfor¬ 
mance.  Accordingly,  the  relative  effects  of  each  training  strategy  on  team  task  completion  time 
and  team  task  accuracy  will  be  quantified.  Furthermore,  to  understand  the  relative  impact  of  each 
training  strategy  on  the  team  processes  that  influence  performance,  ITAE  Program  researchers 
are  designing  an  empirical  study  to  evaluate  three  training  strategies  ( TCT \  CT,  ami  GSCj  in  a 
military-relevant  team  environment.  The  study  will  take  place  within  the  context  of  Unmanned 
Aerial  System  operators  performing  a  close  air  support  (CAS)  mission.  The  approach  used  in  the 
design  of  this  study  is  based  on  the  teamwork  and  team  training  model  summarized  in  Figure  1. 


Figure  1.  High-level  illustration  of  teamwork  and  team  training  model  used  in  study 

Figure  1  suggests  that  team  training  strategies  affect  team  characteristics,  specifically  team¬ 
work  knowledge,  skills,  and  attitudes  (KSAs).  The  changes  in  team  KSAs,  observable  in  team 
processes,  can  be  measured  and  recorded.  Changes  in  team  processes  affect  team  effectiveness, 
which  can  be  recorded  objectively  in  time  and  accuracy  measures. 

Method 

Because  some  studies  have  failed  to  support  the  straightforward  relationship  between  team 
processes  and  outcome  measures  suggested  by  this  model,  the  current  study  will  control  the  va¬ 
riables  that  may  have  influenced  earlier  results.  As  this  project  is  still  in  the  planning  stage,  many 
details  of  the  experimental  design,  apparatus,  and  procedures  have  yet  to  be  developed.  Howev¬ 
er,  the  high-level  experimental  design  and  the  metrics  we  intend  to  examine  will  be  discussed. 

A  between  groups  design  will  be  used  to  examine  the  effects  of  a  single  independent  variable 
(team  training  strategy)  that  will  have  four  levels:  (1)  no  team  training  (control  condition);  (2) 
team  coordination  training;  (3)  cross-training;  and  (4)  guided  self-correction. 

Twenty  four  three-member  teams  will  be  formed  and  randomly  assigned  to  one  of  the  four 
conditions.  These  teams  will  be  trained  as  UAS  operators  in  the  positions  of  pilot,  sensor  opera¬ 
tor,  and  mission  coordinator.  Initial  training  will  be  limited  to  the  individual  position  skills  nec¬ 
essary  to  support  the  close  air  support  (CAS)  role.  Following  this  individual  position  training, 
three  groups  will  receive  teamwork  training,  and  the  control  group  will  receive  none. 

Upon  completion  of  training,  the  teams  will  perform  a  series  of  simulated  UAS  CAS  mis¬ 
sions.  Due  to  the  nature  of  the  task  and  the  information  available  to  the  individual  team  mem¬ 
bers,  the  CAS  task  is  highly  teamwork-interdependent,  and  should  be  fertile  ground  for  observ¬ 
ing  and  recording  team  processes.  Because  the  mission  has  time  and  accuracy  constraints,  objec¬ 
tive  outcome  measures  are  available  and  will  be  recorded. 

Results  of  this  effort  should  allow'  researchers  to  identify  the  relationships  among  team  train¬ 
ing  strategy,  resulting  team  processes,  and  resulting  team  performance.  Such  data  will  provide  a 
first  step  in  identifying  both  algorithms  that  could  be  incorporated  in  IMPRINT  to  predict  team 


performance  moderators  as  a  function  of  team  training  strategy,  and  also  help  provide  guidance 
to  IMPRINT  modelers  on  how  to  best  design  team  task  network  models  to  reflect  team  process 
differences  as  a  function  of  team  training  strategy. 

IMPRINT  Training  Algorithm  Development  for  Maintainers 


Background 

Air  Force  training  for  maintenance  technicians  typically  includes  classroom  instruction  on 
the  characteristics  of  a  system,  which  may  be  instructor-led  or  computer-based.  Students  receive 
instruction  on  the  system’s  theory  of  operation  and  the  instructor  usually  demonstrates  mainte¬ 
nance  procedures.  In  some  cases,  individuals  or  groups  of  students  are  provided  the  opportunity 
to  conduct  hands-on  practice  on  the  actual  weapon  system,  a  simulator,  or  a  part-task  trainer. 

Because  task  practice  is  sometimes  absent  during  training,  instruction  with  and  without 
hands-on  practice  will  be  investigated  in  the  maintenance  training  study  described  in  this  paper. 
Examining  these  instructional  approaches  for  maintenance  will  enable  IMPRINT  modelers  to 
predict  the  effect  they  will  have  on  maintenance  technician  performance  and  how  to  best  allocate 
resources. 

It  is  anticipated  that  this  study  will  provide  significant  insight  into  the  effect  of  instruction  on 
maintainer  performance  by  assessing  the  four  principles  applied  to  task-centered  learning  de¬ 
scribed  by  Merrill  (2001,  2002,  2002a,  2007).  The  four  principles  are:  presentation,  recall,  dem¬ 
onstration  and  application.  Merrill  (in  press)  describes  application  of  these  principles,  as  illus¬ 
trated  in  Table  2. 


Table  2.  Task  Centered-Instructional  Strategies' 


INFORMATION 

PORTRAYAL 

"Tell  “ 

Information 

PRESENTATION 

"Ask" 

Remember 

Information 

RECALL 

“Show” 

Portrayal 

DEMONSTRATION 

"Do"  Apply 
Info  to  por¬ 
trayal 

APPLICATION 

Kinds-of 

Tell  the  definition. 

Recall  the 
definition. 

Show  several  spe¬ 
cific  examples 

Classify  new 
examples. 

How  to 

Tells  the  steps 
and  their  se¬ 
quence. 

Recall  the 
steps  and 
their  se¬ 
quence. 

Show  the  proce¬ 
dure  in  several  dif¬ 
ferent  situations. 

Carry  out  the 
procedure  in 
new  situations 

What 

happens 

Tell  the  condi¬ 
tions  and  conse¬ 
quence  involved 
in  the  process. 

Recall  the 
conditions 
and  conse¬ 
quence  in¬ 
volved  in  the 
process 

Show  the  process 
in  several  different 
situations. 

Predict  a  con¬ 
sequence  or 
find  faulted 
conditions  in 
new  situations. 

The  instruction  received  by  subjects  in  the  maintenance  study  consists  of  varying  the  combi¬ 
nation  of  the  instructional  tactics  “tell,”  “ask,”  “show,”  and  “do.”  The  instructional  strategy  that 
mirrors  the  Air  Force  instructional  approach.  Instruction  with  Practice,  includes  the  tactics  “tell,” 
"ask.”  "show,”  and  “do”  (abbreviated  as  TASD).  The  instructional  strategy  that  represents  the 


1  From  Merrill,  (in  press) 


alternative  Air  Force  instructional  approach.  Instruction  without  Practice,  includes  the  tactics 
“tell,”  “ask/’  and  “show”  (abbreviated  as  TAS). 


Method 

Maintainers  perform  a  variety  of  unique  job  functions,  including  electrical  checks  on  auto¬ 
matic  transmission  components  and  wheel  balancing.  The  following  five  maintenance  tasks  were 
selected  for  the  study  from  these  task  categories:  (1)  measure  solenoid  resistance,  (2)  determine 
if  solenoid  is  faulty,  (3)  remove  wheel  weights,  (4)  determine  wheel  weights  required,  and  (5) 
install  wheel  weights.  The  performance  standard  for  all  tasks  is  accuracy.  Time  was  measured  to 
determine  how  close  to  expert  performance  the  participants  come  during  practice  and  the  trials. 

Fifty  students  participated  for  each  instructional  strategy.  The  “tell”  and  “ask”  components  of 
the  instruction  were  identical  for  all  participants.  In  the  TASD  condition,  participants  were  pro¬ 
vided  a  one  trial  practice  session.  A  different  demonstration  of  the  task  was  provided  to  the  re¬ 
maining  participants.  To  assess  the  extent  of  performance  decay,  participants  returned  for  re¬ 
assessment  60  days  following  initial  training  and  testing. 

A  performance  checklist  was  provided  to  participants  and  researchers  to  accurately  record  the 
errors  made.  Task  elements  performed  correctly  and  ability  to  meet  time  standards  was  also  re¬ 
corded.  During  the  practice  session  of  the  TASD  strategy,  participants  were  permitted  to  ask 
questions  and  receive  feedback  on  each  specific  item  in  the  checklist  during  the  practice  session. 
During  the  testing  phase  of  the  study,  participants  were  provided  feedback  on  their  performance. 

Conclusion 

With  the  completion  of  these  research  efforts,  IMPRINT  users  will  have  a  tool  that  better  es¬ 
timates  how  training  can  impact  system  performance  and  mission  success.  Designing  training  to 
better  support  systems  as  well  as  human  performance  can  reduce  overall  ownership  costs.  En¬ 
hancement  of  the  IMPRINT  training  algorithms  will  further  support  making  trade-off  assess¬ 
ments  between  acquisition  and  training  costs,  adding  features  that  would  reduce  subsequent 
training  versus  deleting  those  features  and  incurring  subsequent  training  penalties.  Prior  to  the 
commencement  of  the  IMPRINT  training  algorithm  enhancement  work,  such  tradeoffs  have  re¬ 
ceived  little  detailed,  explicit  treatment  during  design  evaluations.  The  current  efforts  will  en¬ 
hance  this  capability.  In  cases  where  knowledge  and  skill  demands  require  the  maintenance  of 
proficiency,  the  training  implications  required  to  achieve  desired  levels  of  mission  success  can  be 
estimated,  making  it  easier  to  plan  for  and  justify  operational  unit  training  funds  for  expenditures 
that  assure  adequate  combat  performance  levels  in  personnel  sent  into  theatre.  With  that  kind  of 
modeling  capability,  the  training  community  in  the  operating  commands  can  be  better  supported. 
That  can  save  lives  as  well  as  assure  combat  success. 
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